Control and mapping based on multi-vehicle sensor fusion

ABSTRACT

A sensor data processing and control system acquires data from multiple, disparate sources on a worksite. A plurality of different data pipes are generated for different action systems. The action systems receive data through a corresponding data pipe and generate action signals, based upon aggregated and fused data received through the data pipe.

FIELD OF THE DESCRIPTION

The present description relates to work vehicles. More specifically, thepresent description relates to generating mapping and action signals byaggregating and fusing sensor data from multiple different workmachines.

BACKGROUND

There are a wide variety of different types of work machines. Those workmachines can include such things as loaders, dump trucks, articulatedvehicles, scrapers, excavators, among others. These types of machinesare often deployed on a worksite to perform various operations at theworksite.

Each of these machines may have one or more different sensors deployedon them. For instance, they may have a position sensor (such as a GPSreceiver or other position sensor) that senses a geographic position ofthe vehicle. They may have inertial measurement units (IMUs), cameras(such as backup cameras or other cameras), RADAR or LIDAR systems, amonga variety of other sensors.

In addition, a worksite may have fixed or static sensors that aremounted at the worksite. Such sensors can include cameras, or othersensors that are positioned to sense a desired variable at the worksite.Further, worksites may have unmanned ground vehicles or unmanned aerialvehicles that have sensors on them as well. Those sensors may, forinstance, capture images or other information about the worksite.

The discussion above is merely provided for general backgroundinformation and is not intended to be used as an aid in determining thescope of the claimed subject matter.

SUMMARY

A sensor data processing and control system acquires data from multiple,disparate sources on a worksite. A plurality of different data pipes aregenerated for different action systems. The action systems receive datathrough a corresponding data pipe and generate action signals, basedupon aggregated and fused data received through the data pipe.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter. The claimed subject matter is not limited to implementationsthat solve any or all disadvantages noted in the background.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing one example of a sensor processingarchitecture.

FIG. 2 is a block diagram showing one example of a sensor data fusionsystem in more detail.

FIG. 3 is a flow diagram showing one example of configuration of thearchitecture illustrated in FIG. 1 .

FIG. 4 is a flow diagram illustrating one example of the operation of asensor processing and control system.

FIG. 5 is a block diagram showing one example of the architectureillustrated in FIG. 1 , deployed in a remote server architecture.

FIGS. 6-8 show examples of mobile devices that can be used in thearchitectures shown in the previous FIGS.

FIG. 9 is a block diagram showing one example of a computing environmentthat can be used in the architectures shown in the previous FIGS.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of one example of a sensor processingarchitecture 100. Architecture 100 shows that sensor processing andcontrol system 102 is coupled to various vehicles 104-106 throughnetwork 108. System 102 can also be connected to one or more fixedsensors 110 and any of a wide variety of other systems 112, also throughnetwork 108. Therefore, in one example, network 108 can be a wide areanetwork, a local area network, a near field communication network, acellular network, or any of a wide variety of other networks orcombinations of networks.

The vehicles 104-106 and fixed sensors 110 are illustratively deployedat a worksite 114. The worksite may be a construction site, a quarry, orany of a wide variety of other worksites where vehicles 104-106 are usedto perform various different operations.

Thus, vehicles 104-106 can be any of a wide variety of different typesof vehicles. They can be loaders, bulldozers, dump trucks, articulatedvehicles, excavators, compactors or rollers, backhoes, graders,scrapers, combinations of these and other vehicles (or work machines),etc. Each of the vehicles 104-106 can have a processor 116, data store118, communication system 120, controllable subsystems 122, a set ofsensors 124-126, and any of a wide variety of other items 128. Dependingupon the type of vehicle, controllable subsystems 122 can include suchthings as a propulsion subsystem (such as an engine or other powersource, a drive train, ground-engaging elements, such as wheels ortracks), operating equipment (such as a bucket, a scraper blade), and awide variety of other items. Communication system 120 illustrativelyallows the items on vehicle 106 to communicate with one another, and tocommunicate over network 108. Therefore, communication system 120illustratively facilitates communication over network 108, and it caninclude a controller area network (CAN) communication system or othersystem that allows the items on the vehicle to communicate with oneanother.

Sensors 124-126 can include a wide variety of different types ofsensors. For instance, they can include a position sensor (such as a GPSreceiver or other position sensor that provides a geographic positionand/or pose of vehicle 106), an inertial measurement unit (such as anaccelerometer or other item that senses accelerations imparted onvehicle 106), various different types of cameras, such as a backupcamera, a set of stereo cameras, a camera that is provided withcorresponding logic for sensing bucket volume, a forward looking camera,etc. The cameras can include video cameras or other image capturedevices. The sensors can include RADAR and/or LIDAR or other similartypes of sensors, they can include speed sensors, sensors that sensemachine settings or machine operational parameters, such as fuelconsumption, machine configuration sensors that sense the configurationof the machine on which they are deployed, among a wide variety of othersensors.

In addition, any of vehicles 104-106 can be unmanned aerial vehicles orunmanned ground vehicles. Thus, they can include sensors such as imagecapture devices, video capture devices, or a wide variety of othersensors that can be carried by such vehicles.

Fixed sensors 110 can include any types of sensors that are fixed at aworksite 114. They can, for instance, be cameras, vibration sensors,temperature or soil characteristic sensors, weather sensors, or any of awide variety of other sensors that sense desired characteristics withrespect to worksite 114.

Sensor processing and control system 102 illustratively receives datafrom the sensors at worksite 114. It can receive sensor data from othersystems 112 or access those systems for other reasons.

Sensor processing and control system 102 can also be located on one ormore of the vehicles 104-106 or elsewhere. It is shown as a separatesystem connected over network 108 for the purposes of example only. Itcan be located at a remote server environment (such as in the cloud) orelsewhere as well. Some of those scenarios are described in greaterdetail below.

In the example shown in FIG. 1 , sensor processing and control system102 illustratively includes one or more processors or servers 130, datastore 132, sensor data acquisition system 134, communication system 136,sensor data fusion system 138, action generation system 140, and it caninclude a wide variety of other items 142. Action generation system 140illustratively includes trigger system 144, vehicle settings controlsystem 146, path control system 148, sensor enhancement system 150, sitestatus generation system 152, action configuration/extension interfacelogic 154, and it can include other items 156.

Briefly, in operation, action configuration/extension interface logic154 exposes one or more configuration/extension interfaces 158. A user160 can interact with those interfaces 158 in order to configure variousitems on sensor processing and control system 102. Sensor dataacquisition system 134 acquires sensor data from various sensors 110 and124-126 at worksite 114. It can acquire data from certain sensors ondifferent vehicles 104-106. Sensor data fusion system 138 then fuses orotherwise configures data so that different data pipes (that providedifferent types of data) can be provided to the different action systemsin action generation system 140. For instance, it may be that vehiclesettings control system 146 needs a certain subset of the sensor data inorder to control vehicle settings on the different vehicles 104-106. Atthe same time, it may be that path control system 148 needs a differentsubset of the sensor data in order to generate path control signals tocontrol the path of the different vehicles 104-106. Each of thedifferent action systems 146, 148, 150, 152 and 156 may need its ownsubset of sensor data in order to generate its own action and/or controlsignals. Thus, sensor data fusion system 138 can be configured togenerate different data pipes that provide the different subsets of datato the different action systems in action generation system 140. Sensordata fusion system 138 is described in greater detail below with respectto FIG. 2 .

Each of the action systems 146-156 illustratively includes its owncontrol logic or other action logic in order to perform controloperations or other actions based upon the data received from itscorresponding data pipe. Thus, for instance, vehicle settings controlsystem 146 can include a plurality of different types of settingscontrol logic 162-164. Each item of logic 162-164 can generate differentcontrol signals in order to control different vehicles 104-106,different settings on vehicles 104-106 at worksite 114, etc.

Similarly, path control system 148 may include one or more differentsets of path control logic 166-168. Each item of logic 166-168 can beused to generate control signals to control the steering subsystems onthe various vehicles 104-106 in order to control the path that thosevehicles take, on worksite 114.

Sensor enhancement system 150 can include different sets of enhancementlogic 170-172. Those items of logic can be used to enhance the output ofvarious different types of sensors. For instance, it may be that sensor124 on vehicle 106 is a RADAR sensor that generates an output indicativeof objects in the area surrounding vehicle 106. However, it may be thatvehicle 106 does not have line of sight to different areas on worksite114. In that case, a RADAR sensor on vehicle 104 can be used to augmentthe RADAR sensor on vehicle 106 in order to identify objects on worksite114 that the RADAR sensor on vehicle 106 will not be able to sense. Inanother example, it may be that a fixed camera sensor 110 that istrained to capture images in a particular location on worksite 114 canbe used to augment the RADAR sensor or image capture sensor on vehicle106. It may be, for instance, that fixed sensor 110 can capture imagesin an area where objects cannot be sensed by the sensors on vehicle 106.Thus, the sensor outputs from vehicle 106 can be enhanced using thesensor outputs from fixed sensors 110 or the sensors on vehicle 104,etc. The items of enhancement logic 170-172 can be configured to dothis. They can enhance sensors in a variety of different ways. Sensorsignals from different vehicles can be combined to increase accuracy ofthe sensor output. They can be combined to extend or increase the sensedrange or area of the output, among other things.

Similarly, site status generation system 152 can include one or moredifferent sets of status logic 174-176. Each item of status logic may beconfigured to generate a status output (or status indicator) indicatingthe status of worksite 114. Each status indicator can indicate any of awide variety of different status items. For instance, one item of statuslogic 174-176 may generate an output indicative of the topology ofworksite 114. It can generate multiple different topological maps atdifferent dates so that they can be scrolled through (or otherwiseaccessed) in order to identify, historically, how the topology ofworksite 114 has changed, or is changing, based upon the work being doneby vehicles 104-106.

The items of status logic 174-176 can generate a wide variety of othermapping outputs as well. For instance, they can generate maps of thecurrent and historic paths which vehicles 104-106 have taken on worksite114. This may be useful for many different reasons. The vehicles 104-106may, for instance, be performing soil compaction on worksite 114. Forinstance, where the vehicles are dump trucks and they are traveling overdifferent paths on worksite 114, the dump trucks may be compacting thesoil. This may mean that less compaction will need to be performed laterduring the operations at worksite 114. Tracking the paths of vehicles104-106, historically, along with a variable indicating whether they areloaded or unloaded, can be used by items of logic 174-176 in order tomap soil compaction on worksite 114, even before soil compactors orrollers are deployed. This is just one example of an item of status(soil compaction) that can be generated by site status generation system152.

Each of the systems 146-152 in action generation system 140 thusreceives its own data, from a corresponding data pipe generated bysensor data fusion system 138. It may be that data is continuously beingreceived by the various systems in action generation system 140, or itmay be that those items can receive data, through the corresponding datapipe from sensor data fusion system 138, in response to certaintriggering criteria. Thus, trigger system 144 may include a plurality ofdifferent configurable items of trigger logic 178-180.

Each of the systems 146-152 can have their own triggering criteria, orthe triggering criteria can be shared among them. For instance, it maybe that some systems 146-152 need data continuously and thereforetrigger logic 178 may determine whether sensor data fusion system 138has any new data to provide in the corresponding data pipe. In anotherexample, it may be that one of systems 146-156 only needs to be updatedintermittently, periodically, or when something changes. In that case,the corresponding trigger logic 178-180 will identify when thosetriggering criteria occur, so that data can be obtained through thecorresponding data pipe.

In one example, sensor processing and control system 102 is extendableand configurable. Therefore, action configuration/extension interfacelogic 154 exposes one or more interfaces 158 so that user 160 canconfigure system 102. For instance, user 160 can interact withinterfaces 158 in order to install or configure a different actionsystem in action generation system 140 in addition to, or instead of,those shown. In another example, user 160 can interact with interfaces158 in order to install or configure an item of logic in analready-existing action system. For instance, if user 160 wishes vehiclesettings control system 146 to control a setting on a vehicle 104-106for which no settings control logic already exists, then user 160 caninteract with interfaces 158 in order to install or configure a new itemof settings control logic in vehicle settings control system 146 inorder to control the new setting or group of settings that user 160wishes to be controlled in one or more of the vehicles 104-106. User 160can interact with interfaces 158 in order to install or configuredifferent items of logic in the different action systems 146-156, in asimilar way, in order to extend the functionality of system 140. Actionconfiguration/extension interface logic 154 illustratively exposesinterfaces 158 and detects user interaction with those interfaces andperforms the desired extension or configuration operations in system140, based upon those user interactions.

Before describing the overall operation of architecture 100 in moredetail, one example of a more detailed block diagram of sensor datafusion system 138 (shown in FIG. 2 ) will first be described. In theexample shown in FIG. 2 , it can be seen that sensor data fusion system138 has access to sensor data 182 that has been acquired by sensor dataacquisition system 134. In the example shown in FIG. 2 , sensor datafusion system 138 includes data aggregation system 184 (which, itself,includes different sets of aggregation/packaging logic 186-188),filtering system 190 (which, itself, can include a variety of differentitems of filtering logic 192-194), vehicle settings data pipe generator196, path control data pipe generator 198, sensor enhancement data pipegenerator 200, site status data pipe generator 202, a variety of otherdata pipe generators 204, fusion configuration/extension interface logic206, and it can include other items 208. Fusion configuration/extensioninterface logic 206 exposes interfaces 158 (also shown in FIG. 1 ) sothat user 160 can configure sensor data fusion system 138 in order toprovide different data pipes for the different action systems in actiongeneration system 140.

In doing so, the user can install or configure items ofaggregation/packaging logic 186-188 so that different items of sensordata 182 can be aggregated, as desired, and packaged, as desired, for aparticular data pipe. The user can also configure or install differentitems of filtering logic 192-194 so that the data can be filtered, asdesired, when generating the desired data pipe. It will be noted thatuser 160 can be a vehicle operator who operates one of vehicles 104-106,a construction manager, a user at a separate, remote facility, or any ofa wide variety of other users.

The user can also interact with interfaces 158 in order to install orconfigure different data pipe generators (such as data pipe generators196, 198, 200, 202 and 204). Each data pipe generator illustratively haslogic for selecting a particular data aggregation or data packagegenerated by logic 186-188, and for applying a filter using one or moreitems of filter logic 192-194. It also illustratively has data pipelogic that then arranges the selected, aggregated, and filtered data andprovides it to the corresponding action system in action generationsystem 140 so that the action system can perform its control operationor other action based upon data received from the corresponding datapipe.

Therefore, in the example shown in FIG. 2 , vehicle settings data pipegenerator 196 includes data selector logic 210, filter identifier logic212, data pipe logic 214, and it can include other items 216. Dataselector logic 210 selects the data from one or more packages generatedby aggregation/packaging logic 186-188, which will be used in the datapipe generated by vehicle settings data pipe generator 196. Filteridentifier logic 212 illustratively identifies and applies the filterlogic 192-194 that needs to be applied for the data pipe generated byvehicle settings data pipe generator 196. Data pipe logic 214 thengenerates the data pipe, populates it with data, and makes that dataavailable to the items in action generation system 140 that need it toperform their operations.

Each of the other data pipe generators 198-204 also illustrativelyinclude data selector logic (shown as data selector logic 218, 220, and222), filter identifier logic (shown as filter identifier logic 224, 226and 228), and data pipe logic (shown as data pipe logic 230, 232 and234). Each of the other data pipe generators can also have other items236, 238 and 240 as well.

FIG. 3 is a flow diagram illustrating one example of the operation offusion/configuration extension interface logic 206 and actionconfiguration/extension interface logic 154, in exposing interfaces 158and responding to user interaction with those interfaces.

In the example shown in FIG. 3 , fusion configuration/extensioninterface logic 206 first exposes the interfaces 158 for configuringfusion functionality in sensor data fusion system 138. Exposing thisinterface is indicated by block 250 in the flow diagram of FIG. 3 .

Logic 206 then detects user interactions configuring an item ofaggregation/packaging logic 186-188. This is indicated by block 252.Similar user interactions are detected for configuring items of filterlogic 192-194. This is indicated by block 254. In addition, userinteractions are detected for configuring one or more different datapipe generators 196-204. Detecting these inputs is indicated by block256 in the flow diagram of FIG. 3 . Logic 206 can detect a wide varietyof other configuration inputs as well, and this is indicated by block258 in the flow diagram of FIG. 3 .

Fusion configuration/extension interface logic 206 then installs thelogic that is needed to generate the desired data pipe, if it does notalready exist, or configures it in a way desired by user 160, if it doesalready exist. Installing and/or configuring the data fusionfunctionality to generate the desired data pipe is indicated by block260. In one example, it installs or configures sensoraggregation/packaging logic, as indicated by block 262. In anotherexample, it installs or configures filter logic, as indicated by block264. It can also install or configure data pipe generation logic asindicated by block 266 and/or a wide variety of other items, as indictedby block 268.

Action configuration/extension interface logic 154 can also expose oneor more interfaces 158 in order to configure action generationfunctionality in action generation system 140. This is indicated byblock 270 in the flow diagram of FIG. 3 . For instance, it can detectuser inputs indicative of trigger criteria or trigger logic in triggersystem 144. This is indicated by block 272. It can detect configurationinputs for existing action systems 146-156, or it can detectconfiguration inputs installing or configuring a new action system. Thisis indicated by block 274 in the flow diagram of FIG. 3 . Logic 154 theninstalls the new action system, if it does not yet exist, or configuresan existing action system, based upon the configuration inputs throughinterfaces 158. Installing and/or configuring the trigger and actionsystem logic in action generation system 140 is indicated by block 246in the flow diagram of FIG. 3 .

Once sensor data fusion system 138 and action generation system 140 havebeen configured, then architecture 100 can operate to detect sensor dataacross different vehicles and other sensors at worksite 114 and toperform action or control operations. FIG. 4 is a flow diagramillustrating one example of the operation of architecture 100 in doingthis.

Sensor data acquisition system 134 first detects or acquires sensor datafrom multiple, disparate machines and/or fixed sensors on worksite 114.This is indicated by block 280 in the flow diagram of FIG. 4 . Dataaggregation system 184 (in sensor data fusion system 138) thenaggregates and/or packages the data for the different data pipes thatwill be generated by the data pipe generators in system 138. Aggregatingand/or packaging the data for different data pipes is indicated by block282 in the flow diagram of FIG. 4 .

Filtering system 190 can then perform general filtering on the data.This may be filtering that is done for some or all of the sensor data,such as to filter out outliers, to filter out various noise or otherfiltering. Performing general filtering on the aggregated and/orpackaged data is indicated by block 284.

One or more of the data pipe generators then generate data for thedifferent data pipes for which they were configured. This is indicatedby block 286. In one example, the data pipe generators can be controlledso that they generate data sequentially. For instance, it may be thatvehicle settings data pipe generator 196 is run first followed by theother data pipe generators, in sequence. In another example, all of thedata pipe generators can be running simultaneously to generate the datapipes. The data pipe generators can generate the data pipes asrequested, as needed, or continuously, or in other ways. This isindicated by block 288.

The present discussion will now proceed with respect to vehicle settingsdata pipe generator 196 generating a data pipe for vehicle settingscontrol system 146. This is done by way of example only, and a similardescription could be provided for each of the other data pipe generatorsas well. In the present example, data selector logic 210 then selectsaggregated or packaged data for the vehicle settings data pipe that isto be generated. This is indicated by block 290. Filter identifier logic212 then identifies the particular filter logic 192-194 that is to beapplied to the selected data. It then applies the identified filterlogic to the selected data to obtain filtered data. Identifying andapplying pipe-specific filtering is indicated by block 292 in the flowdiagram of FIG. 4 .

Data pipe logic 214 then generates data, in the data pipe, in a formthat is expected by the vehicle settings control system 146 in actiongeneration system 140. Generating data as expected by the action systemcorresponding to the data pipe is indicated by block 294 in the flowdiagram of FIG. 4 . The data for the data pipes can be generated inother ways as well, and this is indicated by block 296.

At some point, trigger system 144 will detect trigger criteriaindicating that one of the action systems in action generation system140 is to receive data from its corresponding data pipe. Detecting thistrigger is indicated by block 298 in the flow diagram of FIG. 4 . Again,the receipt of data can be triggered when requested by a correspondingaction system, or the corresponding action system can be configured toreceive data continuously, through its data pipe, or the receipt of datacan be based upon other trigger criteria. This is indicated by block 300in the flow diagram of FIG. 4 . Detecting a trigger indicating that datais to be received, through a data pipe, at action generation system 140can be done in a wide variety of other ways as well, and this isindicated by block 302.

Vehicle settings control system 146 then receives data from thecorresponding data pipe that was generated by vehicle settings data pipegenerator 196, and provided by data pipe logic 214. Receiving the datafrom the corresponding data pipe is indicated by block 304 in the flowdiagram of FIG. 4 .

The settings control logic 162-164 then generates an action or controlsignal based upon the received data. This is indicated by block 306. Byway of example, it may be that the data received by the correspondingdata pipe indicates topology or path roughness. In that case, a settingscontrol signal may be generated to reduce the speed of a correspondingvehicle 104-106, when it is approaching an area where the topologicalroughness exceeds a desired threshold. This is just one example of avehicle settings control signal that can be generated, and this isindicated by block 308.

Each of the other action systems in action generation system 140 cangenerate an action and/or control signal based upon the data receivedthrough their corresponding data pipes as well. For instance, pathcontrol system 148 can generate a path control signal that identifies aparticular path for a vehicle. It can then control the steeringsubsystem on that vehicle to follow the desired path. It can also, orinstead, control a user interface display to display a desired path fora driver of the vehicle. Generating path control signals is indicated byblock 310 in FIG. 4 .

Sensor enhancement system 150 can generate sensor enhancement action orcontrol signals to enhance the accuracy or range, or othercharacteristic, of an output from a sensor on a particular vehicle orset of vehicles. Providing a sensor enhancement control signal isindicated by block 312 in the flow diagram of FIG. 4 .

Site status generation system 152 can generate site status control oraction signals as well. For instance, it can generate a map or otherindication indicating how a particular status item corresponding toworksite 114 currently exists, is changing, or has changed over time.These are only examples, and generating site status control signals isindicated by block 314 in the flow diagram of FIG. 4 .

Action generation system 140 can generate action or control signals inother ways as well. This is indicated by block 316 in the flow diagramof FIG. 4 .

The action generation system 140 then applies the signals to perform thedesired action, or control operations, on the vehicles 104-106, in othersystems 112, or elsewhere. Applying the signals to perform the actionand/or control operations is indicated by block 318 in the flow diagramof FIG. 4 .

Architecture 100 continues operation until the operation is complete(e.g., until the operations at worksite 114 have ceased, until variousphases controlled by the action generation system 140 have beencompleted, etc.). Continuing the operation until it is complete isindicated by block 320 in the flow diagram of FIG. 4 .

The present discussion has mentioned processors and servers. In oneexample, the processors and servers include computer processors withassociated memory and timing circuitry, not separately shown. They arefunctional parts of the systems or devices to which they belong and areactivated by, and facilitate the functionality of the other componentsor items in those systems.

Also, a number of user interface displays or other interfaces have beendiscussed. They can take a wide variety of different forms and can havea wide variety of different user actuatable input mechanisms disposedthereon. For instance, the user actuatable input mechanisms can be textboxes, check boxes, icons, links, drop-down menus, search boxes, etc.They can also be actuated in a wide variety of different ways. Forinstance, they can be actuated using a point and click device (such as atrack ball or mouse). They can be actuated using hardware buttons,switches, a joystick or keyboard, thumb switches or thumb pads, etc.They can also be actuated using a virtual keyboard or other virtualactuators. In addition, where the screen on which they are displayed isa touch sensitive screen, they can be actuated using touch gestures.Also, where the device that displays them has speech recognitioncomponents, they can be actuated using speech commands.

A number of data stores have also been discussed. It will be noted theycan each be broken into multiple data stores. All can be local to thesystems accessing them, all can be remote, or some can be local whileothers are remote. All of these configurations are contemplated herein.

Also, the figures show a number of blocks with functionality ascribed toeach block. It will be noted that fewer blocks can be used so thefunctionality is performed by fewer components. Also, more blocks can beused with the functionality distributed among more components.

FIG. 5 is a block diagram of machines 104-106, shown in FIG. 1 , exceptthat they communicate with elements in a remote server architecture 500.In an example, remote server architecture 500 can provide computation,software, data access, and storage services that do not require end-userknowledge of the physical location or configuration of the system thatdelivers the services. In various examples, remote servers can deliverthe services over a wide area network, such as the internet, usingappropriate protocols. For instance, remote servers can deliverapplications over a wide area network and they can be accessed through aweb browser or any other computing component. Software or componentsshown in FIG. 1 as well as the corresponding data, can be stored onservers at a remote location. The computing resources in a remote serverenvironment can be consolidated at a remote data center location or theycan be dispersed. Remote server infrastructures can deliver servicesthrough shared data centers, even though they appear as a single pointof access for the user. Thus, the components and functions describedherein can be provided from a remote server at a remote location using aremote server architecture. Alternatively, they can be provided from aconventional server, or they can be installed on client devicesdirectly, or in other ways.

In the example shown in FIG. 5 , some items are similar to those shownin FIG. 1 and they are similarly numbered. FIG. 5 specifically showsthat sensor processing and control system 102 can be located at a remoteserver location 502. Therefore, machine 104-106 access those systemsthrough remote server location 502.

FIG. 5 also depicts another example of a remote server architecture.FIG. 5 shows that it is also contemplated that some elements of FIG. 1are disposed at remote server location 502 while others are not. By wayof example, data store 132 or other systems 112 can be disposed at alocation separate from location 502, and accessed through the remoteserver at location 502. Regardless of where they are located, they canbe accessed directly by machines 104-106, through a network (either awide area network or a local area network), they can be hosted at aremote site by a service, or they can be provided as a service, oraccessed by a connection service that resides in a remote location.

It will also be noted that the elements of FIG. 1 , or portions of them,can be disposed on a wide variety of different devices. Some of thosedevices include servers, desktop computers, laptop computers, tabletcomputers, or other mobile devices, such as palm top computers, cellphones, smart phones, multimedia players, personal digital assistants,etc.

FIG. 6 is a simplified block diagram of one illustrative example of ahandheld or mobile computing device that can be used as a user's orclient's hand held device 16, in which the present system (or parts ofit) can be deployed. For instance, a mobile device can be deployed inthe operator compartment of machines 104-106 for use in generating,processing, or displaying the stool width and position data. FIGS. 7-9are examples of handheld or mobile devices.

FIG. 6 provides a general block diagram of the components of a clientdevice 16 that can run some components shown in FIG. 1 , that interactswith them, or both. In the device 16, a communications link 13 isprovided that allows the handheld device to communicate with othercomputing devices and in some examples provide a channel for receivinginformation automatically, such as by scanning. Examples ofcommunications link 13 include allowing communication though one or morecommunication protocols, such as wireless services used to providecellular access to a network, as well as protocols that provide localwireless connections to networks.

In other examples, applications can be received on a removable SecureDigital (SD) card that is connected to an interface 15. Interface 15 andcommunication links 13 communicate with a processor 17 (which can alsoembody processors from previous FIGS.) along a bus 19 that is alsoconnected to memory 21 and input/output (I/O) components 23, as well asclock 25 and location system 27.

I/O components 23, in one example, are provided to facilitate input andoutput operations. I/O components 23 for various examples of the device16 can include input components such as buttons, touch sensors, opticalsensors, microphones, touch screens, proximity sensors, accelerometers,orientation sensors and output components such as a display device, aspeaker, and or a printer port. Other I/O components 23 can be used aswell.

Clock 25 illustratively comprises a real time clock component thatoutputs a time and date. It can also, illustratively, provide timingfunctions for processor 17.

Location system 27 illustratively includes a component that outputs acurrent geographical location of device 16. This can include, forinstance, a global positioning system (GPS) receiver, a LORAN system, adead reckoning system, a cellular triangulation system, or otherpositioning system. It can also include, for example, mapping softwareor navigation software that generates desired maps, navigation routesand other geographic functions.

Memory 21 stores operating system 29, network settings 31, applications33, application configuration settings 35, data store 37, communicationdrivers 39, and communication configuration settings 41. Memory 21 caninclude all types of tangible volatile and non-volatilecomputer-readable memory devices. It can also include computer storagemedia (described below). Memory 21 stores computer readable instructionsthat, when executed by processor 17, cause the processor to performcomputer-implemented steps or functions according to the instructions.Processor 17 can be activated by other components to facilitate theirfunctionality as well.

FIG. 7 shows one example in which device 16 is a tablet computer 600. InFIG. 7 , computer 600 is shown with user interface display screen 602.Screen 602 can be a touch screen or a pen-enabled interface thatreceives inputs from a pen or stylus. It can also use an on-screenvirtual keyboard. Of course, it might also be attached to a keyboard orother user input device through a suitable attachment mechanism, such asa wireless link or USB port, for instance. Computer 600 can alsoillustratively receive voice inputs as well.

FIG. 8 shows that the device can be a smart phone 71. Smart phone 71 hasa touch sensitive display 73 that displays icons or tiles or other userinput mechanisms 75. Mechanisms 75 can be used by a user to runapplications, make calls, perform data transfer operations, etc. Ingeneral, smart phone 71 is built on a mobile operating system and offersmore advanced computing capability and connectivity than a featurephone.

Note that other forms of the devices 16 are possible.

FIG. 9 is one example of a computing environment in which elements ofFIG. 1 , or parts of it, (for example) can be deployed. With referenceto FIG. 9 , an example system for implementing some embodiments includesa general-purpose computing device in the form of a computer 810.Components of computer 810 may include, but are not limited to, aprocessing unit 820 (which can comprise processors from previous FIGS.),a system memory 830, and a system bus 821 that couples various systemcomponents including the system memory to the processing unit 820. Thesystem bus 821 may be any of several types of bus structures including amemory bus or memory controller, a peripheral bus, and a local bus usingany of a variety of bus architectures. Memory and programs describedwith respect to FIG. 1 can be deployed in corresponding portions of FIG.9 .

Computer 810 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 810 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media is different from, anddoes not include, a modulated data signal or carrier wave. It includeshardware storage media including both volatile and nonvolatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical disk storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store the desired information and which canbe accessed by computer 810. Communication media may embody computerreadable instructions, data structures, program modules or other data ina transport mechanism and includes any information delivery media. Theterm “modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal.

The system memory 830 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 831and random access memory (RAM) 832. A basic input/output system 833(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 810, such as during start-up, istypically stored in ROM 831. RAM 832 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 820. By way of example, and notlimitation, FIG. 9 illustrates operating system 834, applicationprograms 835, other program modules 836, and program data 837.

The computer 810 may also include other removable/non-removablevolatile/nonvolatile computer storage media. By way of example only,FIG. 9 illustrates a hard disk drive 841 that reads from or writes tonon-removable, nonvolatile magnetic media, an optical disk drive 855,and nonvolatile optical disk 856. The hard disk drive 841 is typicallyconnected to the system bus 821 through a non-removable memory interfacesuch as interface 840, and optical disk drive 855 are typicallyconnected to the system bus 821 by a removable memory interface, such asinterface 850.

Alternatively, or in addition, the functionality described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Application-specific Integrated Circuits (e.g., ASICs),Application-specific Standard Products (e.g., ASSPs), System-on-a-chipsystems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 9 , provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 810. In FIG. 9 , for example, hard disk drive 841 isillustrated as storing operating system 844, application programs 845,other program modules 846, and program data 847. Note that thesecomponents can either be the same as or different from operating system834, application programs 835, other program modules 836, and programdata 837.

A user may enter commands and information into the computer 810 throughinput devices such as a keyboard 862, a microphone 863, and a pointingdevice 861, such as a mouse, trackball or touch pad. Other input devices(not shown) may include a joystick, game pad, satellite dish, scanner,or the like. These and other input devices are often connected to theprocessing unit 820 through a user input interface 860 that is coupledto the system bus, but may be connected by other interface and busstructures. A visual display 891 or other type of display device is alsoconnected to the system bus 821 via an interface, such as a videointerface 890. In addition to the monitor, computers may also includeother peripheral output devices such as speakers 897 and printer 896,which may be connected through an output peripheral interface 895.

The computer 810 is operated in a networked environment using logicalconnections (such as a local area network—LAN, or wide area network WAN)to one or more remote computers, such as a remote computer 880.

When used in a LAN networking environment, the computer 810 is connectedto the LAN 871 through a network interface or adapter 870. When used ina WAN networking environment, the computer 810 typically includes amodem 872 or other means for establishing communications over the WAN873, such as the Internet. In a networked environment, program modulesmay be stored in a remote memory storage device. FIG. 10 illustrates,for example, that remote application programs 885 can reside on remotecomputer 880.

It will be noted that the above discussion has described a variety ofdifferent systems, components and/or logic. It will be appreciated thatsuch systems, components and/or logic can be comprised of hardware items(such as processors and associated memory, or other processingcomponents, some of which are described below) that perform thefunctions associated with those systems, components and/or logic. Inaddition, the systems, components and/or logic can be comprised ofsoftware that is loaded into a memory and is subsequently executed by aprocessor or server, or other computing component, as described below.The systems, components and/or logic can also be comprised of differentcombinations of hardware, software, firmware, etc., some examples ofwhich are described below. These are only some examples of differentstructures that can be used to form the systems, components and/or logicdescribed above. Other structures can be used as well.

It should also be noted that the different examples described herein canbe combined in different ways. That is, parts of one or more examplescan be combined with parts of one or more other examples. All of this iscontemplated herein.

Example 1 is a vehicle control system, comprising:

a sensor data acquisition system that receives sensor data from a sensoron a plurality of different work machines at a worksite and thatgenerates an acquired data signal indicative of the received sensordata;

a sensor data fusion system that receives the acquired data signal andgenerates a plurality of different data pipes, each data pipe providingdifferent corresponding sensor data, based on the acquired data signal;and

an action generation system comprising a plurality of action systems,each action system being coupled to a different one of the plurality ofdifferent data pipes and generating an action signal based on thecorresponding sensor data.

Example 2 is the vehicle control system of any or all previous exampleswherein each of the plurality of different action systems comprises:

a plurality of items of action signal generation logic, each item ofaction signal generation logic generating a different action signalbased on the sensor data corresponding to the data pipe to which it iscoupled.

Example 3 is the vehicle control system of any or all previous exampleswherein the sensor data fusion system comprises:

a vehicle settings data pipe generator configured to generate a vehiclesettings data pipe that provides vehicle settings sensor data based onthe acquired data signal.

Example 4 the vehicle control system of any or all previous exampleswherein the action generation system, as one of the plurality of actionsystems, comprises:

a vehicle settings control system that receives the vehicle settingsdata from the vehicle settings data pipe and generates a vehiclesettings control signal to control a setting on one of the plurality ofdifferent work machines based on the vehicle settings data

Example 5 is the vehicle control system of any or all previous exampleswherein the vehicle settings control system comprises:

a plurality of different items of settings control logic, eachgenerating a different settings control signal based on the vehiclesettings data.

Example 6 is the vehicle control system of any or all previous exampleswherein the sensor data fusion system comprises:

a path control data pipe generator configured to generate a path controldata pipe that provides path control sensor data based on the acquireddata signal.

Example 7 is the vehicle control system of any or all previous exampleswherein the action generation system, as one of the plurality of actionsystems, comprises:

a path control system that includes a plurality of different items ofpath control logic, each generating a different path control signal tocontrol a path on a different one of the plurality of different workmachines based on the path control sensor data.

Example 8 is the vehicle control system of any or all previous exampleswherein the sensor data fusion system comprises:

a sensor enhancement data pipe generator configured to generate a sensorenhancement data pipe that provides sensor enhancement sensor data basedon the acquired data signal.

Example 9 is the vehicle control system of any or all previous exampleswherein the action generation system, as one of the plurality of actionsystems, comprises:

a sensor enhancement system that includes a plurality of different itemsof enhancement logic, each generating a different sensor enhancementsignal to enhance accuracy of a different sensor on the plurality ofdifferent work machines based on the sensor enhancement data.

Example 10 is the vehicle control system of any or all previous exampleswherein the sensor data fusion system comprises:

a site status data pipe generator configured to generate a site statusdata pipe that provides site status data based on the acquired datasignal.

Example 11 is the vehicle control system of any or all previous exampleswherein the action generation system, as one of the plurality of actionsystems, comprises:

a site status generation system that includes a plurality of differentitems of status logic, each generating a different site status signal togenerate a site status indicator indicative of a different site statusvariable based on the site status data.

Example 12 is the vehicle control system of any or all previous examplesand further comprising:

action interface logic configured to expose a configuration/extensioninterface and detect user interactions with the configuration/extensioninterface to add or configure one of the items of action signalgeneration logic to generate a different action signal.

Example 13 is the vehicle control system of any or all previous exampleswherein the sensor data fusion system comprises:

fusion interface logic configured to expose a fusionconfiguration/extension interface and detect user interactions with thefusion configuration/extension interface to add or configure a data pipegenerator to generate a different data pipe.

Example 14 is a method of generating a control signal, comprising:

receiving sensor data from a sensor on a plurality of different workmachines at a worksite;

generating an acquired data signal indicative of the received sensordata;

generating a plurality of different data pipes, each data pipe providingdifferent corresponding sensor data, based on the acquired data signal,by selectively aggregating and filtering the received sensor data foreach data pipe; and

generating an action signal with each of a plurality of action systems,each action system being coupled to a different one of the plurality ofdifferent data pipes, based on the corresponding sensor data.

Example 15 is the method of any or all previous examples whereingenerating an action signal comprises:

generating a different action signal with each of a plurality of itemsof action signal generation logic, based on the sensor datacorresponding to the data pipe to which the item of action signalgeneration logic is coupled.

Example 16 is the method of any or all previous examples whereingenerating a plurality of data pipes comprises:

generating a vehicle settings data pipe that provides vehicle settingssensor data based on the acquired data signal;

generating a path control data pipe that provides path control sensordata based on the acquired data signal;

generating a sensor enhancement data pipe that provides sensorenhancement sensor data based on the acquired data signal; and

generating a site status data pipe that provides site status data basedon the acquired data signal.

Example 17 is the method of any or all previous examples whereingenerating a different action signal comprises:

receiving the vehicle settings data from the vehicle settings data pipe;and

generating a vehicle settings control signal to control a setting on oneof the plurality of different work machines based on the vehiclesettings data.

Example 18 is the method of any or all previous examples whereingenerating a different action signal comprises:

receiving the path control sensor data from the path control data pipe;and

generating a different path control signal to control a path of adifferent one of the plurality of different work machines based on thepath control sensor data.

Example 19 is the method of any or all previous examples whereingenerating a different action signal comprises:

receiving the sensor enhancement sensor data from the sensor enhancementdata pipe;

receiving the status data from the site status data pipe;

generating a sensor enhancement signal to enhance accuracy of a sensoron the plurality of different work machines based on the sensorenhancement data; and

generating a site status signal to generate a site status indicatorindicative of a site status variable based on the site status data.

Example 20 is an extendable and configurable control system, comprising:

a sensor data acquisition system that receives sensor data from a sensoron a plurality of different work machines at a worksite and thatgenerates an acquired data signal indicative of the received sensordata;

a sensor data fusion system that receives the acquired data signal andgenerates a plurality of different data pipes, each data pipe providingdifferent corresponding sensor data, based on the acquired data signal;

fusion interface logic configured to expose a fusionconfiguration/extension interface and detect user interactions with thefusion configuration/extension interface to add and configure a datapipe generator to generate each of the plurality of different datapipes;

an action generation system comprising a plurality of action systems,each action system being coupled to a different one of the plurality ofdifferent data pipes and generating a corresponding action signal basedon the corresponding sensor data; and

action interface logic configured to expose a configuration/extensioninterface and detect user interactions with the configuration/extensioninterface to add and configure one of the action systems to generate thecorresponding action signal.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. A vehicle control system, comprising: one or moreprocessors; and a data store that stores a set of computer executableinstructions that, when executed by the one or more processors,configure the one or more processors to implement: a sensor dataacquisition system that: receives respective sensor data from each of aplurality of different sensors, each sensor of the plurality of sensorsbeing on a different one of a plurality of work machines at a worksite,the respective sensor data from each sensor being indicative of a sameworksite characteristic; and generates a plurality of acquired datasignals indicative of the received respective sensor data; a sensor datafusion system that: receives the plurality of acquired data signals andpackages the plurality of respective sensor data to form a plurality ofpackaged data subsets, each packaged data subset of the plurality ofpackaged data subsets being different than the other packaged datasubsets of the plurality of packaged data subsets; and generates aplurality of different data pipes, each data pipe of the plurality ofdifferent data pipes being configured to receive one of the packageddata subsets of the plurality of packaged data subsets; and an actiongeneration system configured to generate a map of the worksiteindicative of the worksite characteristic at different locations in theworksite based on the plurality of respective sensor data, the actiongeneration system comprising: a plurality of action systems, each actionsystem being coupled to a different one of the plurality of differentdata pipes and configured to generate an action signal based on thepackaged data subset received in the data pipe and apply the actionsignal to control one or more of the plurality of different workmachines to perform an action based on the action signal.
 2. The vehiclecontrol system of claim 1 wherein each of the plurality of differentaction systems comprises: a plurality of items of action signalgeneration logic, each item of action signal generation logic generatinga different action signal based on the packaged data subsetcorresponding to the data pipe to which it is coupled.
 3. The vehiclecontrol system of claim 2 wherein the sensor data fusion systemcomprises: a vehicle settings data pipe generator configured to generatea vehicle settings data pipe that provides vehicle settings sensor databased on the respective sensor data.
 4. The vehicle control system ofclaim 3 wherein the action generation system, as one of the plurality ofaction systems, comprises: a vehicle settings control system thatreceives the vehicle settings data from the vehicle settings data pipeand generates a vehicle settings control signal to control a setting onone of the plurality of different work machines based on the vehiclesettings data.
 5. The vehicle control system of claim 4 wherein thevehicle settings control system comprises: a plurality of differentitems of settings control logic, each generating a different settingscontrol signal based on the vehicle settings data.
 6. The vehiclecontrol system of claim 2 wherein the sensor data fusion systemcomprises: a path control data pipe generator configured to generate apath control data pipe that provides path control sensor data based onthe respective sensor data.
 7. The vehicle control system of claim 6wherein the action generation system, as one of the plurality of actionsystems, comprises: a path control system that includes a plurality ofdifferent items of path control logic, each generating a different pathcontrol signal to control a path on a different one of the plurality ofdifferent work machines based on the path control sensor data.
 8. Thevehicle control system of claim 2 wherein the sensor data fusion systemcomprises: a sensor enhancement data pipe generator configured togenerate a sensor enhancement data pipe that provides sensor enhancementsensor data based on the respective sensor data.
 9. The vehicle controlsystem of claim 8 wherein the action generation system, as one of theplurality of action systems, comprises: a sensor enhancement system thatincludes a plurality of different items of enhancement logic, eachgenerating a different sensor enhancement signal to enhance accuracy ofa different sensor on the plurality of different work machines based onthe sensor enhancement data.
 10. The vehicle control system of claim 2wherein the sensor data fusion system comprises: a site status data pipegenerator configured to generate a site status data pipe that providessite status data based on the respective sensor data.
 11. The vehiclecontrol system of claim 10 wherein the action generation system, as oneof the plurality of action systems, comprises: a site status generationsystem that includes a plurality of different items of status logic,each generating a different site status signal to generate a site statusindicator indicative of a different site status variable based on thesite status data.
 12. The vehicle control system of claim 2, wherein theset of computer executable instructions, when executed by the one ormore processors, further configure the one or more processors toimplement: action interface logic configured to expose aconfiguration/extension interface and detect user interactions with theconfiguration/extension interface to add or configure one of the itemsof action signal generation logic to generate a different action signal.13. The vehicle control system of claim 2 wherein the sensor data fusionsystem comprises: fusion interface logic configured to expose a fusionconfiguration/extension interface and detect user interactions with thefusion configuration/extension interface to add or configure a data pipegenerator to generate a different data pipe.
 14. A method of generatinga control signal, comprising: receiving respective sensor data from eachof a plurality of different sensors, each sensor of the plurality ofsensors being on a different one of a plurality of work machines at aworksite, the respective sensor data from each sensor being indicativeof a same worksite characteristic; generating a plurality of acquireddata signals indicative of the received respective sensor data;generating a plurality of different data pipes, each data pipe providinga different set of the respective sensor data by selectively aggregatingand filtering the received respective sensor data for each data pipe;generating a map of the worksite based on the respective sensor datafrom the plurality of different work machines, wherein generating themap comprises generating a site status map indicative of the sameworksite characteristic at different locations in the worksite;generating an action signal with each of a plurality of action systems,each action system being coupled to a different one of the plurality ofdifferent data pipes, based on the respective sensor data; and applyingthe action signal to control at least one of the plurality of differentwork machines.
 15. The method of claim 14 wherein receiving therespective sensor data comprises receiving, from each sensor on each ofthe plurality of different work machines, vehicle pathing dataindicative of paths traveled by each different work machine at theworksite from a first time to a current time.
 16. The method of claim 15wherein generating the site status map comprises generating the sitestatus map indicating current and historical paths for each differentwork machine at the worksite from the first time to a current time basedon the vehicle pathing data.
 17. The method of claim 15 whereinreceiving the respective sensor data further comprises receiving vehicleload data indicative of a load carried by each of the different workmachines and wherein generating the site status map comprises generatingthe site status map indicative of a change in compaction of theworksite.
 18. The method of claim 14 wherein generating the site statusmap comprises generating the site status map indicative of a change in atopography of the worksite from a first time to a current time.
 19. Anextendable and configurable control system, comprising: one or moreprocessors; and a data store that stores computer executableinstructions that, when executed by the one or more processors,configure the one or more processors to implement: a sensor dataacquisition system that receives sensor data from a sensor on aplurality of different work machines at a worksite and that generates anacquired data signal indicative of the received sensor data, the sensordata including vehicle load data indicative of a load carried by each ofthe different work machines; a sensor data fusion system that receivesthe acquired data signal and generates a plurality of different datapipes, each data pipe providing different corresponding sensor data,based on the acquired data signal; fusion interface logic configured toexpose a fusion configuration/extension interface and detect userinteractions with the fusion configuration/extension interface to addand configure a data pipe generator to generate each of the plurality ofdifferent data pipes; a site status generation system configured togenerate a site status map indicative of a change in compaction of theworksite; an action generation system comprising a plurality of actionsystems, each action system being coupled to a different one of theplurality of different data pipes and generating a corresponding actionsignal based on the corresponding sensor data; and action interfacelogic configured to expose a configuration/extension interface anddetect user interactions with the configuration/extension interface toadd and configure one of the action systems to generate thecorresponding action signal to control the plurality of different workmachines.